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Displaying a wide field of view video image 
Field Of The Present Invention 

The present invention relates to wide field imaging. 
Background Of The Present Invention 

Systems that seek to acquire very wide field of view (e.g. 
3 60 2 ) images are significant for a number of applications in 
both still and motion picture capturing and display. One such 
system employs a camera rotating around the horizontal of its 
focal plane used for capturing still panoramic photographs. 
Many images are taken, ranging from of the order of about 7 for 
consumer photographs to more than 100 for professional 
panoramas. The camera is typically mounted on a tripod, with 
the pan axis centered on the horizontal center of the focal 
plane. Parallax errors are reduced as the number of images is 
increased. 

Sophisticated software has been recently developed that 
allows a consumer-quality panorama to be built from a series of 
handheld images. However, such techniques are based on 
panoramic still images, and cannot produce video panoramas since 
the product of exposure time and the number of images that must 
be taken exceeds the time per frame of a typical video sequence. 

In other systems, a single camera coupled with a domed, 
spherical, or toroidal section mirror is used. The camera is 
usually mounted in such systems above the mirror so that the 
camera can see a e.g. 3 60 2 surround band around the mirror. The 
mirror may be placed on a conference room table to provide a 
view of everyone sitting around a meeting table or may be placed 
on a tripod for panoramic landscape pictures . 
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In yet other systems, multiple mirrors and cameras are 
utilized to acquire the panoramic views. One such system uses 6 
cameras looking down on mirrors arranged in a ring. The mirrors 
for this system consist of a 6-side pyramid, with different 
cameras looking down on each of the mirrors. The ring of 
cameras is supported by a post at the center of the pyramid of 
mirrors . 

Summary Of The Present Invention 

In one aspect of the present invention provides a system 
for displaying a wide field of view video image of a location. 
The system comprises a plurality of location cameras for 
placement at the location. The location cameras capture the wide 
field of view video image as a plurality of individual video 
images that together cover the desired field of view. The system 
also comprises a distance sensor unit. The distance sensor unit 
senses distances of closest objects in one or more overlap areas 
between field of views of the neighboring location cameras. The 
system further comprises a display unit for displaying the 
plurality of individual video images to a user for creating a 
visual experience of the location based on the sensed distances 
to the closest object . 

Brief Description Of The Drawings 

Fig. 1A is a schematic drawing illustrating a front view of 
a surrogate device in an embodiment of the present invention; 

Fig. IB is a schematic drawing illustrating a top view of 
the surrogate device of Fig. 1A; 

Fig. 1C is a schematic drawing illustrating a detail of 
Fig. IB for one object configuration; 

Fig. ID is a schematic drawing illustrating a detail of 
Fig . IB for another object configuration; 



specif ication20031 3764.doc 

200313764 



3 

Fig. 2. is a schematic drawing illustrating a display 
system in an embodiment of the present invention; 

Fig. 3 is a schematic drawing illustrating a parallax 
artifact ; 

Fig. 4 is a schematic drawing illustrating a parallax 
artifact; 

Fig. 5 is a flowchart illustrating a method of displaying a 
wide field of view video image in an embodiment of the present 
invention . 

Detailed Description Of The Embodiments 

The system of the example embodiment consists of two main 
parts, a display system at the user's location and a remotely 
located surrogate device. 

The surrogate device is connected to the display system via 
high-speed networking in the example embodiment, and stands in 
the place of the user at the remote location. 

Fig. 1A shows a schematic drawing of the surrogate device 
100 of the example embodiment. The surrogate device 100 has a 
"head" 102 made from four display panels, e.g. 104 in a square, 
outwardly facing arrangement. Live video of the user's face 106 
is displayed on all four sides, namely on all four LCD panels of 
the surrogate device 100. 

Four cameras e.g. 108, 110 in the corners of the surrogate 
devices head 102 together capture 3 60° surround live video from 
the remote location at which the surrogate device 10 0 is 
located. The cameras e.g. 108, 110 are positioned near the mid- 
point of the height of the displays e.g. 104 in the example 
embodiment, to achieve a proximity to the eye-level of the 
displayed user face 106. 

Four directional microphones e.g. 112, 114 are also 
provided in the corners of the surrogate devices head 102 for 
capturing the remote sound field. The surrogate device head 102 
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also comprises speakers e.g. 116, 118 in the corners of the 
surrogate device ' s head 102 for transmitting a sound field from 
the user's location at the remote location. 

The surrogate device 100 further comprises an internal 
computer unit 12 0 for connecting to the cameras and microphones, 
processing data, and sending it to the display system. The 
computer unit 120 also receives live video of the user's head 
106 and displays it on the four display screens e.g. 104. The 
surrogate device also receives audio signals from the user, and 
then processes and outputs the signals to the speakers e.g. 116, 
118. 

The video data from the four cameras, e.g. 108, 110 is 
compressed and transmitted over a high-speed network (not shown) 
to computers at the location of the display system. An antenna 
121 is utilized in the example embodiment for wireless 
connection to the network. 

Fig. IB shows a schematic drawing of a top view of the 
surrogate device 100 . This drawing illustrates how the four 
cameras , e.g. 108, 110 together capture a 3 60 - field of view 
around the surrogate devices head 102 . Obj ects are located 
around the surrogate 100, inc luding obj ects 124 to 127 at 
different distances in overlap regions 128 to 131 of field of 
views of adjacent cameras. 

Fig . 2 is a schematic drawing of one example of a display 
system 200 that can be used in connection with the surrogate 
device 100. The video from the surrogate device 100 (see Fig. 
1A) is projected on the walls/screens e.g. 202 of a display cube 
204 by four projectors e.g. 206. The projectors e.g. 206 are 
housed in "hush boxes" e.g. 208 to make them virtually inaudible 
in the example embodiment. Speakers e.g. 210 are mounted above 
and below each projection screen e.g. 2 02 of the display cube 
204. 
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The hush boxes e.g. 2 08 in the example embodiment are built 
using sound isolation techniques familiar to a person skilled in 
the art. Each box e.g. 2 08 has a double-pane window 212 for the 
projector light beam to shine out of the box 2 08 and onto the 
screen 2 02. Each box e.g. 2 08 also has measures for adequately 
cooling the projectors 206. 

Computers 214, 215 are placed in an adjacent room 216 for 
sound isolation purposes, and drive the projectors e.g. 2 06 and 
the speakers e.g. 210 with video and audio information 
transmitted from the surrogate device 100 (see Fig. 1) at the 
remote location via network 217. The network 217 of the present 
embodiment includes an antenna 218 to enable wireless 
communication with the surrogate 10 0, via its respective antenna 
121. 

Anechoic foams are placed on the walls of the room 218 in 
the example embodiment, for eliminating local reflections. 
Stray reflection of the light is reduced and the contrast of the 
display screens e.g. 2 02 is increased by using black anechoic 
foams . 

The user 22 0 is located inside the display cube 2 04, which 
includes a suitable entrance arrangement (not shown) in at least 
one of the walls e.g. 202. The images on the projection screens 
e.g. 2 02 are presented in life size in the example embodiment. 
This means that the angle sub- tended by objects on the screen is 
about the same angle as if the user 220 was actually at the 
remote location viewing the objects themselves. It will be 
appreciated by a person skilled in the art that the reproduction 
is not exact unless the user's head is centered in the display 
cube 204. Life size presentation of both local and remote 
participants in the example embodiment is advantageous for 
preserving the gaze of the participants. 

Cameras e.g. 222 are provided in each corner of the display 
cube 204 which take video images of the user 220 for display on 
the display panels e.g. 104 of the surrogate device 100 (see 
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Fig. 1A) . The user 22 0 wears a wireless lapel microphone (not 
shown) . A receiver 224 connected to one of the computers 214 
receives the audio data from the lapel microphone in the example 
embodiment. The video and audio data from the cameras e.g. 222 
and the wireless microphone is transmitted over the high-speed 
network 217 via a wireless communications channel to the 
surrogate device 100 (see Fig. 1A) . 

In the following, it will be described how parallax 
artifacts are automatically corrected in the example embodiment. 
Because the cameras e.g. 108, 110 in the surrogate device's head 
102 (see Fig. 1) cannot have a common optical center at the same 
level as the display of the user's eyes on the display panels 
e.g. 104 (see Fig. 1A) , parallax results. This parallax causes 
visibility gaps and/or duplication of imagery between adjacent 
images. For example, consider a geometry 300 as illustrated in 
Fig. 3, where the cameras 3 02, 3 04 have a 90-degree field of 
view. If a person 306 at the remote location stands between the 
two cameras 3 02, 3 04, the user cannot see them. 

Consider instead a geometry 400 shown in Fig. 4, where the 
field of view of the cameras 402, 404 is widened to 143 2 . Here, 
the user will see two copies of a person 406 standing in the 
overlap region 408 of the field of views of the cameras 402, 
404, one projected on each of adjacent display wall at the 
user's location. While this is an improvement over not seeing 
the person at all, it is still less than desirable in most 
situations. For example, if the two copies of the person are 
presented, this introduces errors in preserving the gaze of both 
the user and the remote person. 

It has been recognized by the inventors that if an object 
at the remote location is located at exactly the point where the 
outermost rays captured by the cameras intersect, the user will 
see only one copy of that object, projected in the area where 
adjacent projection screens come together. 
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Thus, partial automatic correction of parallax artifacts 
can be enabled in an example embodiment of the present 
invention, by adjusting the horizontal span of the displayed 
images from each camera depending on the nearest distance to an 
object at the edges of each camera's field of view. One copy of 
the closest object or person is projected for the user 
independent of the distance of the closest object or person from 
the surrogate device in the example embodiment. 

Returning now to Fig. IB, this is accomplished in the 
example embodiment by measuring the distance from the surrogate 
device 100 to objects e.g. 124 in the overlap area 128 of the 
vertical edges of the fields of view of cameras 108, 110 and 
modifying the horizontal field of view of the projected images 
so that they intersect at the distance of the closest obj ect in 
that overlap area 12 8 . 

It will be appreciated that the distance to objects in the 
overlap area of two adjacent cameras can be acquired using a 
number of methods, one of which will be discussed below for the 
example embodiment . 

The example embodiment utilizes infrared sensors e.g. 132 
of a type that sends out a narrow beam 13 4 and based on the 
distance of the obj ect it reflects from, the beam illuminates 
different pixels in a linear array of photocells (not shown) . 
The corresponding distance can be read out over a serial bus by 
the computer unit 120 (see Fig. 1A) . 

A pair of infrared sensors e.g. 132, 13 6 is used in the 
example embodiment for each camera e.g. 108, which send out 
narrow beams 13 4, 13 8 along respective field of view boundary 
lines 140, 142. The beams 134, 138 are directed downwardly to 
substantially follow the bottom left and right corners of the 
extending field of view of the camera 108. 

Four additional infrared sensors e.g. 144 are provided in 
the example embodiment and are positioned centrally below the 
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displays e.g. 104 of the surrogate device 100 (see Fig. 1A) . 
Each of the four additional sensors e.g. 144 sends out a narrow 
beam 146 which extends horizontally and centrally through the 
overlap area 128 of two cameras 108, 110. The distance to the , 
nearest object is computed as the minimum distance reported by 
all the sensors monitoring the area between a pair of cameras. 

Once the distance to the nearest object in the overlap area 
is known, the horizontal field of view of a display of images 
from the cameras adjacent to the overlap area can be modified. 
Figs. 1C and ID illustrate two different scenarios for adjusting 
the displayed horizontal field of view depending on closest 
objects in the overlap region 131 between adjacent cameras 148, 
150. 

Turning initially to Fig. 1C, the closest object 126 in 
that scenario is located on the left side of the overlap region 
131. In this scenario, vertical edges 152, 154 of the displayed 
images from cameras 148 and 150 respectively are chosen such 
that they intersect at point 156 at the measured object distance 
on a line extending centrally through the overlap region 131, 
which coincides with the narrow beam 158 from one of the 
distance sensors. It will be appreciated that thus, the object 
12 6 will be visible substantially only on the image originating 
from camera 148, but not on the image originating from camera 
150. 

In the scenario shown in Fig. ID, the object 12 6 is 
centrally located within the overlap region 131. Here, the 
vertical edges 152b and 154b of the displayed images originating 
from cameras 148 and 150 respectively are chosen to intersect at 
point 156b. In this geometry, the shortest distance to the 
object 126 is detected by the narrow beam 156 of the central 
sensor, which can be used directly to determine the intersection 
point 156b. 

It will be appreciated that respective "halves" of the 
object 126 will be visible in the images from cameras 148 and 
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150 respectively, thus avoiding undesired duplication of the 
object 12 6 in the displayed wide field of view video image. 

There are four cameras and a 3 60-degree surround view is 
synthesized in the example embodiment. The vertical field of 
view displayed should therefore preferably be the same in all 
four images. If the vertical field of view were not the same 
throughout, there would be discontinuities where parts of an 
object appeared taller or shorter than other parts. 

Having a uniform vertical field of view displayed means 
that any changes to the horizontal field of view displayed must 
be made without affecting the vertical field of view - i.e., 
anamorphically . Also, the distance to the nearest object in 
each of the four overlap areas 128 to 131 around the surrogate 
device 100 will usually be different from each other. The 
modification of each projected image is preferably a function of 
the nearest distance in both of its adjacent overlap areas. 

As distances to the closest objects change during the 
capture of the individual video images, the horizontal scaling 
may be continually adjusted in real-time, if desired. It will 
be appreciated that a number of different approaches to 
modifying the horizontal scale can be employed. Once approach 
will be described below for the example embodiment. The pixel 
column of the desired left and right edges of the imagery on the 
screen is calculated, and a single horizontal scale is computed. 
In this approach scale transition artifacts would be limited to 
the seam between screens, which already has a number of 
artifacts due to the screen discontinuity. Any artifacts from a 
transition in a horizontal scale should be less evident there. 
The desired left and right edges of the projected image can be 
found from trigonometry. 

Fig. 5 shows a flowchart 500 illustrating a method for 
displaying a wide field of view video image embodying the 
present invention. At step 502, a plurality of cameras 504, 506 
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are used to take individual video images 505, 507 of a location 
508, together covering a desired wide field of view. 

In step 510, distances of the nearest object 512 at the 
overlap area the adjacent edges of the individual field of views 
of the cameras 504, 506 to each of the cameras 504, 506 is 
measured. 

In step 514, the distance information is utilized to 
anamorphically adjust the horizontal span of the individual 
displayed images 516, 518 to form a composite wide field of view 
image 52 0. In the composite image 52 0, one half of the object 
508 is displayed in each partial image 516, 518 respectively. 

It will be appreciated by the person skilled in the art 
that numerous modifications and/or variations may be made to the 
present invention as shown in the specific embodiments without 
departing from the spirit or scope of the invention as broadly 
described. The present embodiments are, therefore, to be 
considered in all respects to be illustrative and not 
restrictive . 

For example, while infrared sensors have been used in the 
described embodiment to sense the distances of the nearest 
objects, it will be appreciated that the present invention can 
be implemented utilizing a variety of different sensing 
techniques, including e.g. ultrasonic sensors or stereo vision 
technique sensor implementations. 

Furthermore, while the adjusting of the horizontal span of 
the displayed images in the example embodiment includes 
computing a single horizontal scale, in different embodiments 
the horizontal scale could be modified by e.g. dividing the 
screen in half and modifying the horizontal scaling of each half 
screen. In the region between the half screens, the horizontal 
scale could e.g. be gradually changed, to reduce scale 
transition artifacts at the seam between the half screens. 
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It will also be appreciated that the present invention is 
not limited to the use of four cameras to capture the desired 
wide field of view video image. Rather, the number of cameras 
can be chosen to suit a particular desired coverage of the wide 
field of view, both horizontally and vertically, and also in 
terms of depth coverage between adjacent field of views of the 
individual cameras . 

Similarly, the number and configuration of distance sensor 
elements can be chosen in different embodiments to achieve a 
desired coverage and depth resolution. 



